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DETAILED ACTION 
Introduction 

1 . This communication is in response to the Applicants' communication dated 
January 26, 2007. Claims 1, 20-26, 30 and 31 were canceled. Claims 2-13, 16-19 and 
27-29 were amended. Claims 32-35 were added. Claims 2-19, 27-29 and 32-35 of the 
application are pending. 

Color Drawings 

2. The petition to Approve color drawings dated November 2, 2004 has been approved. 

Examiner's Amendment 

3. Authorization for this examiner's amendment was given in a telephone conversation by 
Mr. Steven Lawrenz on April 11, 2007. 

An examiner's amendment to the record appears below. Should the changes and/or 
additions be unacceptable to the applicants, an amendment may be filed as provided by 37 CFR 
1 .3 12. To ensure consideration of such an amendment, it MUST be submitted no later than the 
payment of the issue fee. 



Application/Control Number: 10/692,020 
Art Unit: 2123 



Page 3 



4. In the specification: 

Page 1 0, Para 0044, Line 4, "the feature points to be extract from" 
has been changed to 

— the feature points to be extracted from ~. 

Page 17, Para 0067, Lines 2-3, "The graph 800 is encoded using a key 810" 
has been changed to 

~ The graph 800 is encoded using a key 801 --. 

5. In the claims: 
Cancel claim 6. 
Replace claims 8-1 1 with: 

8. A method in a computing system for tracking movement of a surface having an 
arbitrary appearance relative to a camera, comprising: 

capturing an image of the surface using the camera; 

analyzing the captured image of the surface to identify visual features present in the 
captured image; 

from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed based 
upon a comparison of levels of contrast provided by each of the identified features; 
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receiving a sequence of images captured by the camera, at least some of which constitute 
a view of at least a portion of the surface; 
for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3-dimensional location and orientation of the surface in the 
image of the sequence with respect to the camera; 

using the determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are relative to those 
of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

9. A method in a computing system for tracking movement of a surface having an 
arbitrary appearance relative to a camera, comprising: 

capturing an image of the surface using the camera; 

analyzing the captured image of the surface to identify visual features present in the 
captured image; 

from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed based 
upon a comparison of levels of uniqueness of each of the identified features; 
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receiving a sequence of images captured by the camera, at least some of which constitute 
a view of at least a portion of the surface; 
for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3 -dimensional location and orientation of the surface in the 
image of the sequence with respect to the camera; 

using the determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are relative to those 
of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

10. The method of claim 1 1 wherein the selection of visual features is performed 
based upon a comparison of levels of accuracy with which they can be used to determine the 
location and the orientation of the surface, 

11. A method in a computing system for tracking moyement of a surface having an 
arbitrary appearance relative to a camera, comprising: 

capturing an image of the surface using the camera; 

analyzing the captured image of the surface to identify visual features present in the 
captured image; 
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from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed based 
upon a comparison of levels of contrast provided by each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which constitute 
a view of at least a portion of the surface; 

for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3 -dimensional location and orientation of the surface in the 
image of the sequence with respect to the camera, 

wherein the selection of visual features includes selecting visual features in at 
least two different size ranges, the method further comprising selecting one of the size ranges 
based upon a measure of a distance to the surface's present location, and 

wherein the selected features in the selected size range are used to determine the 
location and the orientation of the surface; 

using the determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are relative to those 
of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

Amended claim 13, Line 5, "fiducial markers to determining" 
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has been changed to 
fiducial markers for determining — . 

Replace claim 16 with: 

1 6. The method of claim 9 wherein the 3-dimensional location and orientation of the 
surface in the image of the sequence is determined without use of explicit fiducial markers. 

Replace claims 18-19 with: 

1 8. The method of claim 9 wherein the movement of the surface is tracked within an 
environment, and wherein, between two successive images of the sequence, the camera moves 
relative to the environment. 

19. The method of claim 9 wherein the movement of the surface is tracked within an 
environment, and wherein, between two successive images of the sequence, the surface moves 
relative to the environment. 

Replace claim 28 with: 

28. A method in a,computing system for determining 3-dimensional location and 
orientation of a subject surface in a distinguished perspective image of the subject surface, the subject 
surface having innate visual features, comprising: 

identifying a plurality of subsets of innate visual features of the subject surface, each subset 
containing visual features of a different general size; 
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based on an earlier-determined 3-dimensional location of the subject surface, selecting one 
subset of visual features; 

using locations of the visual features of the selected subset in a perspective image of the 
subject surface that precedes the distinguished perspective image in time, and identifying search 
zones in the distinguished perspective image; 

searching the identified search zones for the visual features of the selected subset to 
determine 2-dimensional positions at which the visual features of the selected subset occur; 

based on the determined 2-dimensional positions, determining the 3-dimensional 
location and orientation of a subject surface in the distinguished perspective image; 

using the determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are relative to those 
of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

Replace claims 32-35 with: 

32. A computer-readable storage medium comprising computer executable 
instructions which when executed on a computing system cause the computing system to 
perform a method for tracking movement of a surface having an arbitrary appearance relative to 
a camera, the method comprising: 

capturing an image of the surface using the camera; 

analyzing the captured image of the surface to identify visual features present in the 
captured image; 
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from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed based 
upon a comparison of levels of contrast provided by each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which constitute 
a view of at least a portion of the surface; 

for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3 -dimensional location and orientation of the surface in the 
image of the sequence with respect to the camera; 

using the determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are relative to those 
of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

33. A computer-readable storage medium comprising computer executable 
instructions which when executed on a computing system cause the computing system to 
perform a method for tracking movement of a surface having an arbitrary appearance relative to 
a camera, the method comprising: 

capturing an image of the surface using the camera; 
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analyzing the captured image of the surface to identify visual features present in the 
captured image; 

from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed based 
upon a comparison of levels of uniqueness of each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which constitute 
a view of at least a portion of the surface; 

for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3-dimensional location and orientation of the surface in the 
image of the sequence with respect to the camera; 

' using the determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are relative to those 
of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

34. A computer-readable storage medium comprising computer executable 
instructions which when executed on a computing system cause the computing system to 
perform a method for tracking movement of a surface having an arbitrary appearance relative to 
a camera, the method comprising: 
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capturing an image of the surface using the camera; 

analyzing the captured image of the surface to identify visual features present in the 
captured image; 

from the identified visual features, selecting a plurality of visual features for use in 
tracking the movemisnt of the surface, wherein the selection of visual features is performed based 
upon a comparison of levels of contrast provided by each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which constitute 
a view of at least a portion of the surface; 

for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3-dimensional location and orientation of the surface in the 
image of the sequence with respect to the camera, 

wherein the selection of visual features includes selecting visual features in at least 
two different size ranges, the method fiirther comprising selecting one of the size ranges based 
upon a measure of a distance to the surface*s present location, and 

wherein the selected features in the selected size range are used to determine the 
location and the orientation of the surface; 

using the determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are relative to those 
of the surface to obtain an augmented sequence of images; and 
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displaying the augmented sequence of images using a computer display. 

35. A computer-readable storage medium comprising computer executable 

i 

instructions which when executed on a computing system cause the computing system to 
perform a method for determining 3 -dimensional location and orientation of a subject surface in a 
distinguished perspective image of the subject surface, the subject surface having innate visual 
features, comprising: 

identifying a plurality of subsets of innate visual features of the subject surface, each subset 
containing visual features of a different general size; 

based on an earlier-determined 3-dimensional location of the subject surface, selecting one 
siibset of visual features; 

using locations of the visual features of the selected subset in a perspective image of the 
subject surface that precedes the distinguished perspective image in time, and identifying search 
zones in the distinguished perspective image; 

searching the identified search zones for the visual features of the selected subset to 
determine 2-dimensional positions at which the visual features of the selected subset occur; 

based on the determined 2-dimensional positions, determining the 3-dimensional 
location and orientation of a subject surface in the distinguished perspective image; 

using the determined location and orientation of the surface to introduce a supplemental 
image into the images of the sequence at a size, location, and orientation that are relative to those 
of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 
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A clean copy of allowed claims is attached. 

Reasons for Allowance 

6. Claims 2-5, 7-19, 27-29 and 32-35 of the application are allowed over prior art of record. 

7. The following is an Examiner's statement of reasons for the indication of allowable 
subject matter: 

The closest prior art of record shows: 

(1) machine vision systems for tracking the movement of multiple objects within a 
predetermined area or volume; the systems are capable of creating three-dimensional tracking 
information using a system of markers placed upon the objects to be tracked; the markers are 
followed by an overlapping configuration of tracking cameras; as the image information is 
analyzed from the camera's two-dimensional view, it is combined to create 3D coordinates of 
each marker as the marker moves about in the designated tracking volume; based on the detected 
marker 3D locations and the pre-known relationship between the markers and objects, the system 
is able to re-assemble any given object's movement (Aman et aL, U.S. Patent Application 
2003/0095186); 

(2) method and apparatus for generating representations of objects in the three- 
dimensional computer model using moving pictures such as video images; image data fi-om 
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cameras having different views of objects in the scene is processed to determine whether image 
data identified for an object is associated with more than one object; a three-dimensional 
computer model is rendered for a chosen viewing direction; image data for an object in the scene 
is processed to produce a three dimensional model of the object in dependence upon surface 
planes of the object identified fi-om the image data; image data from at least two cameras having 
different views of an object is processed to identify planar surfaces of the object on which feature 
points lie and to represent the object in three-dimensional manner in dependence upon the 
identified planes; a three-dimensional computer model is rendered for the chosen viewing 
direction and information indicating the accuracy of the model generated (Rowe et aL, U. S, 
Patent 6,914,599); and 

(3) a vision based mixed reality system using natural feature points contained in images 
captured by a pair of stereo cameras in conjunction with pre-defined fixed fiducial markers; the 
system first uses pre-defined fiducial markers to estimate a projection matrix between real and 
virtual coordinate systems; at the same time, the system picks up and tracks a set of natural 
feature points from the initial image; as the user moves around in the MR environment, the initial 
markers fall out of the camera frame and the natural features are then used to recover a 
projection matrix; augmented reality produces an environment in which virtual objects are 
superimposed on user's view of the real environment; virtual objects should be superimposed on 
the right place as if they really exist in the real world; the method estimates the position and 
orientation of user's view point from images captured by the cameras (Kabbara et al., "A stereo 
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vision based mixed reality system with natural feature point tracking", Nara Institute of Science 
and technology, March 2001). 

Additional state of the art reviewed and considered by the Examiner is found in Izquierdo 
et al., "Modehng arbitrary objects based on geometric surface conformity", IEEE, 1999; Lin, C, 
"Invariants of three-dimensional contours", IEEE, 1988; Kobbelt et al., "Feature sensitive surface 
extraction from volume data", ACM, 2001; Blanz et al., "A morphable model for the synthesis of 
3D faces", ACM 1999. 

None of these references taken either alone or in combination with the prior art of record 
discloses a method in a computing system for tracking movement of a surface having an 
arbitrary appearance relative to a camera, specifically including: 

(Claim 8) "fi*om the identified visual features, selecting a plurality of visual features for 
use in tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of levels of contrast provided by each of the identified features". 

None of these references taken either alone or in combination with the prior art of record 
discloses a method in a computing system for tracking movement of a surface having an 
arbitrary appearance relative to a camera, specifically including: 

(Claim 9) "from the identified visual features, selecting a plurality of visual features for 
use in tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of levels of uniqueness of each of the identified features". 
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None of these references taken either alone or in combination with the prior art of record 
discloses a method in a computing system for tracking movement of a surface having an 
arbitrary appearance relative to a camera, specifically including: 

(Claim 11) "from the identified visual features, selecting a plurality of visual features 
for use in tracking the movement of the surface, wherein the selection of visual features is 
performed based upon a comparison of levels of contrast provided by each of the identified 
features". 

None of these references taken either alone or in combination with the prior art of record 
discloses a method in a computing system for determining 3-dimensional location and orientation of 
a subject surface in a distinguished perspective image of the subject surface, specifically including: 

(Claim 28) "identifying a plurality of subsets of innate visual features of the subject surface, 
each subset containing visual features of a different general size; 

based on an earlier-determined 3-dimensional location of the subject surface, selecting one 
subset of visual features; 

using locations of the visual features of the selected subset in a perspective image of the 
subject surface that precedes the distinguished perspective image in time, and identifying search 
zones in the distinguished perspective image". 

None of these references taken either alone or in combination with the prior art of record 
discloses a computer-readable storage medium comprising computer executable 
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instructions which when executed on a computing system cause the computing system to 
perform a method for tracking movement of a surface having an arbitrary appearance relative to 
a camera, specifically including: 

(Claim 32) "from the identified visual features, selecting a plurality of visual features for 
use in tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of levels of contrast provided by each of the identified features". 

None of these references taken either alone or in combination with the prior art of record 
discloses a computer-readable storage medium comprising computer executable instructions 
which when executed on a computing system cause the computing system to perform a method 
for tracking movement of a surface having an arbitrary appearance relative to a camera, 
specifically including: 

(Claim 33) "from the identified visual features, selecting a plurality of visual features 
for use in tracking the movement of the surface, wherein the selection of visual features is 
performed based upon a comparison of levels of uniqueness of each of the identified features". 

None of these references taken either alone or in combination with the prior art of record 
discloses a computer-readable storage medium comprising computer executable instructions 
which when executed on a computing system cause the computing system to perform a method 
for tracking movement of a surface having an arbitrary appearance relative to a camera, 
specifically including: 
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(Claim 34) "from the identified visual features, selecting a plurality of visual features 
for use in tracking the movement of the surface, wherein the selection of visual features is 
performed based upon a comparison of levels of contrast provided by each of the identified 
features". 

None of these references taken either alone or in combination with the prior art of record 
discloses a computer-readable storage medium comprising computer executable instructions 
which when executed on a computing system cause the computing system to perform a method 
for determining 3-dimensional location and orientation of a subject surface in a distinguished 
perspective image of the subject surface, specifically including: 

(Claim 35) "identifying a plurality of subsets of innate visual features of the subject surface, 
each subset containing visual features of a different general size; 

based on an earUer-determined 3-dimensional location of the subject surface, selecting one 
subset of visual features; 

using locations of the visual features of the selected subset in a perspective image of the 
subject surface that precedes the distinguished perspective image in time, and identifying search 
zones in the distinguished perspective image". 

8. Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the 
issue fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 
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9. Any inquiry concerning this comnnunication or earlier communications from the 
examiner should be directed to Dr. Kandasamy Thangavelu whose telephone number is 
571-272-371 7. The examiner can nonnally be reached on IVIonday through Friday from 
8:00 AlVI to 5:30 PIVI. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Paul Rodriguez, can be reached on 571-272-3753. The fax phone number 
for the organization where this application or proceeding Is assigned is 571-273-8300. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to TC 2100 Group receptionist: 571-272-2100. 

Infomnation regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-21 7-91 97 (toll-free). ^ ~p /\/ X • 
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2. The method of claim 8 wherein the identification of the 2-dimensional 
positions at which the selected features occur in the image is predicated on an assumption that 
the selected features appear as coplanar in the image. 

3. The method of claim 8 wherein the surface is a 2-dimensional surface. 

4. The method of claim 8 wherein the surface is a flat surface. 

5. The method of claim 8 wherein the surface is an irregular body that appears 
flat when observed at a distance. 

6. Canceled. 

7. The method of claim 8, further comprising using the determined location and 
orientation of the surface to superimpose a view of a 3-dimensional object over the surface. 

8. A method in a computing system for tracking movement of a surface having 
an arbitrary appearance relative to a camera, comprising: 

capturing an image of the surface using the camera; 

analyzing the captured image of the surface to identify visual features present in the 
captured image; 
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from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of levels of contrast provided by each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which 
constitute a view of at least a portion of the surface; 

for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3-dimensional location and orientation of the surface in 
the image of the sequence with respect to the camera; 

using the determined location and orientation of the surface to introduce a 
supplemental image into the images of the sequence at a size, location, and orientation that 
are relative to those of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

9, A method in a computing system for tracking movement of a surface having 
an arbitrary appearance relative to a camera, comprising: 
capturing an image of the surface using the camera; 

analyzing the captured image of the surface to identify visual features present in the 
captured image; 



2 



from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of levels of uniqueness of each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which 
constitute a view of at least a portion of the surface; 

for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3-dimensional location and orientation of the surface in 
the image of the sequence with respect to the camera; 

using the determined location and orientation of the surface to introduce a 
supplemental image into the images of the sequence at a size, location, and orientation that 
are relative to those of the surface to obtain an augmented sequence of images; and 
displaying the augmented sequence of images using a computer display. 

10. The method of claim 1 1 wherein the selection of visual features is performed 
based upon a comparison of levels of accuracy with which they can be used to determine the 
location and the orientation of the surface. 

11. A method in a computing system for tracking movement of a surface having 
an arbitrary appearance relative to a camera, comprising: 

capturing an image of the surface using the camera; 
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analyzing the captured image of the surface to identify visual features present in the 
captured image; 

from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of levels of contrast provided by each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which 
constitute a view of at least a portion of the surface; 
for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3-dimensional location and orientation of the surface in 
the image of the sequence with respect to the camera, 

wherein the selection of visual features includes selecting visual features in at 
least two different size ranges, the method further comprising selecting one of the size ranges 
based upon a measure of a distance to the surface's present location, and 

wherein the selected features in the selected size range are used to determine 
the location and the orientation of the surface; 

using the determined location and orientation of the surface to introduce a 
supplemental image into the images of the sequence at a size, location, and orientation that 
are relative to those of the surface to obtain an augmented sequence of images; and 
displaying the augmented sequence of images using a computer display. 
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12. The method of claim 9, further comprising, for each image of the sequence: 
based upon the 2-dimensional positions at which the features are identified in one or 

more prior images of the sequence, predicting 2-dimensional positions at which the features 
will occur in the current image; and 

establishing search zones about the predicted positions, and wherein identifying the 2- 
dimensional positions at which the selected features occur in the image comprises searching 
the established search zones for the selected features. 

13. The method of claim 9 wherein the surface, in addition to having an arbitrary 
appearance, is modified to contain one or more fiducial markers, 

the method fiirther comprising, in a first image of the sequence, identifying and 
analyzing a portion of the image corresponding to the fiducial markers for determining the 3- 
dimensional location and orientation of the surface in the image, and wherein the determining 
the 3-dimensional location and orientation of the surface in the first image of the sequence is 
used to identify the 2-dimensional positions at which the selected features occur in the first 
image of the sequence. 

14. The method of claim 13 wherein each fiducial marker comprises a unique 
distinguishing pattem. 

15. The method of claim 1 3 wherein each fiducial marker is square-shaped. 
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1 6. The method of claim 9 wherein the 3-dimensional location and orientation of the 
surface in the image of the sequence is determined without use of expUcit fiducial markers. 

17. The method of claim 9 fiirther comprising capturing the sequence of images 
using the camera, and wherein the determination is made in real-time with respect to the 
capture. 

18. The method of claim 9 wherein the movement of the surface is tracked within 
an environment, and wherein, between two successive images of the sequence, the camera 
moves relative to the environment. 

19. The method of claim 9 wherein the movement of the surface is tracked within 
an environment, and wherein, between two successive images of the sequence, the surface 
moves relative to the environment. 

27. The mettiod of claim 28 wherein the selected innate visual features of the subject 
surface number at least four. 

28. A method in a computing system for determining 3-dimensional location and 
orientation of a subject surface in a distinguished perspective image of the subject surface, the 
subject surface having innate visual features, comprising: 

identifying a plurality of subsets of innate visual features of the subject surface, each 
subset containing visual features of a different general size; 
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based on an earlier-detennined 3-dimensional location of the subject surface, selecting 
one subset of visual features; 

using locations of the visual features of the selected subset in a perspective image of the 
subject surface that precedes the distinguished perspective image in tinie, and identifying search 
zones in the distinguished perspective image; 

searching the identified search zones for the visual features of the selected subset to 
determine 2-dimensional positions at which the visual features of the selected subset occur; 

based on the determined 2-dimensipnal positions, determining the 3-dimensional 
location and orientation of a subject surface in the distinguished perspective image; 

using the determined location and orientation of the surface to introduce a 
supplemental image into the images of the sequence at a size, location, and orientation that 
are relative to those of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

29. The method of claim 28 wherein the determination of the 2-dimensional locations 
at which the selected visual features occur is predicated on an assumption that the selected visual 
features appear as coplanar in the image. 

32. A computer-readable storage medium comprising computer executable 
instructions which when executed on a computing system cause the computing system to 
perform a method for tracking movement of a surface having an arbitrary appearance relative 
to a camera, the method comprising: 

capturing an image of the surface using the camera; 
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analyzing the captured image of the surface to identify visual features present in the 
captured image; 

from the identified visual features, selecting a pluraUty of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of leyels of contrast provided by each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which 
constitute a view of at least a portion of the surface; 

for each image of the sequence: . 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3-dimensional location and orientation of the surface in 
the image of the sequence with respect to the camera; 

using the determined location and orientation of the surface to introduce a 
supplemental image into the images of the sequence at a size, location, and orientation that 
are relative to those of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

33. A computer-readable storage medium comprising computer executable 
instructions which when executed on a computing system cause the computing system to 
perform a method for tracking movement of a surface having an arbitrary appearance relative 
to a camera, the method comprising: 

capturing an image of the surface using the camera; 
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analyzing the captured image of the surface to identify visual features present in the 
captured image; 

from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of levels of uniqueness of each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which 
constitute a view of at least a portion of the surface; 

for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3-dimensional location and orientation of the surface in 
the image of the sequence with respect to the camera; 

using the determined location and orientation of the surface to introduce a 
supplemental image into the images of the sequence at a size, location, and orientation that 
are relative to those of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 

34. A computer-readable storage mediimi comprising computer executable 
instructions which when executed on a computing system cause the computing system to 
perform a method for tracking movement of a surface having an arbitrary appearance relative 
to a camera, the method comprising: 

capturing an image of the surface using the camera; 
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analyzing the captured image of the surface to identify visual features present in the 
captured image; 

from the identified visual features, selecting a plurality of visual features for use in 
tracking the movement of the surface, wherein the selection of visual features is performed 
based upon a comparison of levels of contrast provided by each of the identified features; 

receiving a sequence of images captured by the camera, at least some of which 
constitute a view of at least a portion of the surface; 

for each image of the sequence: 

identifying 2-dimensional positions at which the selected features occur in the 

image; and 

based upon the 2-dimensional positions at which the selected features are 
identified in the image, determining 3 -dimensional location and orientation of the surface in 
the image of the sequence with respect to the camera, 

wherein the selection of visual features includes selecting visual features in at 
least two different size ranges, the method further comprising selecting one of the size ranges 
based upon a measure of a distance to the surface's present location, and 

wherein the selected features in the selected size range are used to determine 
the location and the orientation of the surface; 

using the determined location and orientation of the surface to introduce a 
supplemental image into the imagies of the sequence at a size, location, and orientation that 
are relative to those of the surface to obtain an augmented sequence of images; and 
displaying the augmented sequence of images using a computer display. 
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35. A computer-readable storage medium comprising computer executable 
instructions which when executed on a computing system cause the computing system to 
perform a method for determining 3-dimensional location and orientation of a subject surface in 
a distinguished perspective image of the subject surface, the subject surface having innate visual 
features, comprising: 

identifying a plurality of subsets of innate visual features of the subject surface, each 
subset containing visual features of a different general size; 

based on an earlier-determined B^dimensional location of the subject surface, selecting 
one subset of visual features; 

using locations of the visual features of the selected subset in a perspective image of the 
subject surface that precedes the distinguished perspective image in time, and identifying search 
zones in the distinguished perspective image; 

searching the identified search zones for the visual features of the selected subset to 
detemiine 2-dimensional positions at which the visual features of the selected subset occur; 

based on the determined 2-dimensional positions, determining the 3-dimensional 
location and orientation of a subject surface in the distinguished perspective image; 

using the determined location and orientation of the surface to introduce a 
supplemental image into the images of the sequence at a size, location, and orientation that 
are relative to those of the surface to obtain an augmented sequence of images; and 

displaying the augmented sequence of images using a computer display. 
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